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INTRODUCTION 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  program  was 
chartered  by  the  Office  of  the  Secretary  of  Defense  in  October  1994  to  investigate  the  utility  of 
advanced  distributed  simulation  (ADS)  technologies  for  support  of  test  and  evaluation  (T&E). 
Among  other  things,  the  charter  tasked  JADS  to  “identify  the  critical  constraints,  concerns,  and 
methodologies  when  using  ADS  for  T&E.”1  JADS  has  gained  considerable  experience  in 
planning  and  conducting  distributed  tests  and  is  using  that  experience  to  develop  methodologies 
which  will  be  passed  as  legacy  products  to  the  T&E  community. 

A  previous  paper2  described  a  general  test  planning  methodology  which  incorporated  excursion 
loops  for  the  examination  of  ADS  alternatives.  The  steps  are  used  to  assist  testers  in  deciding  if 
ADS-based  testing  makes  sense  for  specific  T&E  applications.  An  example  of  a  case  in  which 
ADS  implementation  is  appropriate  would  be  for  the  mission-level  evaluation  of  a  system  in  which 
the  system  under  test  (SUT)  interacts  with  a  number  of  other  systems  and  man-in-the-loop 
interactions  are  involved.  In  this  case,  a  live  test  involving  a  large  number  of  players  would  not  be 
affordable,  and  using  a  digital  system  model  (DSM)  for  the  entire  scenario  would  not  produce 
credible  results  for  the  human  interactions  and  decision-making  processes. 

This  paper  outlines  a  methodology  developed  by  JADS  for  implementing  ADS-based  T&E,  once 
the  decision  has  been  made  to  use  ADS.  The  implementation  methodology  follows  the  steps 
given  in  the  Defense  Modeling  and  Simulation  Office  (DMSO)  High  Level  Architecture  (HLA) 
Federation  Development  and  Execution  Process  (FEDEP)  model3  and  amplifies  them  by  adding 
lessons  learned  from  JADS  testing  experience.  Key  methodology  activities  discussed  are  the 
careful  determination  of  test  objectives  and  all  appropriate  requirements  before  design  of  the  ADS 
architecture  begins.  Also,  the  importance  of  integration  testing  is  emphasized.  The  methodology 
is  designed  to  take  the  tester  through  all  aspects  of  ADS-based  test  planning,  designing, 
development/construction,  check-out,  execution,  and  reporting. 

FEDEP  MODEL 

The  FEDEP  model  groups  the  activities  needed  to  develop  and  execute  a  distributed  test  into  six 
steps: 

•  Step  1 :  The  test  sponsor  or  evaluator  and  the  distributed  test  development  team  define  and 
agree  on  a  set  of  objectives  and  document  what  must  be  accomplished  to  achieve  those 
objectives. 
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•  Step  2:  A  representation  of  the  real  world  domain  of  interest  is  developed  and  described  in 
terms  of  a  set  of  required  objects  and  interactions. 

•  Step  3:  Distributed  test  participants  (federates)  are  determined,  and  required  functionalities 
are  allocated  to  the  participants. 

•  Step  4:  The  Federation  Object  Model  (FOM)  is  developed  (if  HLA  is  implemented), 
participant  agreements  on  consistent  databases/algorithms  are  established,  and  modifications 
to  federates  are  implemented  (as  required). 

•  Step  5:  All  necessary  distributed  test  implementation  activities  are  performed,  and  testing  is 
conducted  to  ensure  interoperability  requirements  are  being  met. 

•  Step  6:  The  distributed  test  is  executed,  outputs  are  generated,  and  results  provided. 

The  FEDEP  model  breaks  the  six  steps  into  activities,  as  shown  in  Table  1. 


Table  1.  Distributed  Test  Planning  and  Implementation  Activities. 


STEP 

ACTIVITIES 

1 .  Define  Distributed  Test  Objectives 

1.1  Identify  Needs 

1.2  Develop  Objectives 

2.  Develop  Conceptual  Model 

2.1  Develop  Scenario 

2.2  Perform  Conceptual  Analysis 

2.3  Develop  Test  Requirements 

3.  Design  Distributed  Test 

3.1  Select  Participants 

3.2  Allocate  Functionality 

3.3  Prepare  Plan 

4.  Develop  Distributed  Test 

4.1  Develop  FOM 

4.2  Establish  Participant  Agreements 

4.3  Implement  Participant  Modifications 

5.  Integrate  and  Test  Architecture 

5.1  Plan  Execution 

5.2  Integrate  and  Test  ADS  Architecture 

6.  Execute  Distributed  Test  and  Analyze 
Results 

6.1  Execute  Distributed  Test 

6.2  Process  Output 

6.3  Prepare  Results 

The  test  implementation  methodology  presented  here  uses  the  FEDEP  model  as  a  framework  and 
amplifies  the  activities  using  lessons  learned  from  JADS  testing  experience.  The  full  methodology 
is  documented  in  a  JADS  special  report.4  This  paper  will  focus  on  the  following  key  activities  as 
an  illustration  of  the  overall  methodology: 

•  Activity  2.3  Develop  Test  Requirements 

•  Activity  3.1  Select  Participants 

•  Activity  3.2  Allocate  Functionality 

•  Activity  3.3  Prepare  Plan 
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•  Activity  4.2  Establish  Participant  Agreements 

•  Activity  5.1  Plan  Execution 

•  Activity  5.2  Integrate  and  Test  ADS  Architecture 

DEVELOP  TEST  REQUIREMENTS 

Before  the  distributed  test  architecture  can  be  designed,  detailed  test  requirements  must  be 
defined.  These  requirements  should  be  based  on  the  distributed  test  objectives,  should  be  directly 
testable,  and  provide  the  implementation  level  guidance  needed  to  design  and  develop  the 
distributed  test.  The  test  requirements  will  also  be  the  basis  for  the  criteria  for  evaluating  test 
results.  Major  top-level  requirements  which  should  be  addressed  include  the  following: 

•  Fidelity  requirements.  The  fidelity  requirements  for  all  players  represented  in  the  distributed 
test  scenarios  must  be  determined.  The  required  fidelity  depends  upon  the  maturity  of  the 
SUT,  the  SUT  test  objectives,  and  the  nature  of  the  interactions  between  the  SUT  and  the 
other  players. 

•  Interaction  requirements.  Determine  the  data  types  which  must  be  exchanged  among  players 
to  permit  interactions,  including  entity  state  data,  tactical  messages,  launch  and  detonation 
indications  (if  appropriate),  and  trial  start  and  stop  notification. 

•  Latency  requirements.  Determine  the  maximum  acceptable  latency  and  latency  variations  for 
each  pair  of  interacting  players.  The  maximum  latency  requirement  will  be  determined  by  how 
closely  coupled  the  interactions  are  and  by  the  maximum  allowable  error  in  the  location  of  one 
player  as  perceived  by  the  other 

•  Data  reliability  requirements.  Determine  the  maximum  acceptable  level  of  ADS-induced 
errors,  such  as  dropout  rate  and  out-of-order  data  messages.  The  allowable  errors  may  vary 
with  data  types. 

•  Data  analysis  requirements.  Draft  a  preliminary  data  management  and  analysis  plan  (DMAP) 
which  details  the  analysis  approach  for  each  test  objective.  From  the  DMAP  determine  which 
data  must  be  collected  and  the  analysis  techniques  to  be  applied. 

After  all  these  requirements  have  been  developed,  the  capability  of  the  support  agencies  (e.g., 
simulation  or  range  facilities,  networking  and  engineering  team)  to  support  the  test  must  be 
clearly  stated  and  documented,  such  as  by  a  Statement  of  Capability  (SOC).  The  SOC  documents 
the  set  of  requirements  and  provides  a  clear  statement  of  the  support  agency’s  capabilities, 
constraints,  and  limitations  in  meeting  those  requirements. 

The  support  agencies  also  need  to  create  an  integrated,  detailed  work  breakdown  structure 
(WBS)  early  in  the  program  which  is  consistent  with  the  SOC.  It  is  also  important  to  have 
accurate  cost  estimates  allocated  against  the  WBS  tasks  in  order  to  help  program  management 
decisions. 

SELECT  PARTICIPANTS 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  determine  the  suitability  of 
individual  player  representations  (e.g.,  simulations,  HWIL  labs,  or  live  players/ranges)  to  become 
participants  in  the  distributed  test.  The  input  to  this  activity  is  the  conceptual  model  (developed  in 
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Activity  2.2)  which  is  a  description  of  the  players,  their  actions,  and  any  interactions  between 
players  that  need  to  be  included  in  the  distributed  test. 

This  selection  is  driven  primarily  by  the  following  factors: 

•  perceived  ability  of  potential  representations  to  represent  the  players’  behavior  and  the 
interactions  specified  in  the  conceptual  model. 

•  fidelity  requirements  for  each  player. 

•  managerial  constraints,  such  as  availability,  cost,  schedule,  and  security  considerations. 

•  technical  constraints,  such  as  VV&A  status  and  portability. 

•  For  live  players,  the  selection  of  particular  test  ranges  is  also  driven  by  considerations  of  range 
instrumentation  quality  and  quantity  and  data  processing  capability. 

ALLOCATE  FUNCTIONALITY 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  allocate  the  responsibility  to 
represent  the  entities  and  actions  in  the  conceptual  model  to  the  participants.  This  activity  will 
allow  for  the  assessment  of  whether  the  set  of  selected  participants  provides  the  full  set  of 
required  functionality  or  whether  one  or  more  of  the  representations  will  need  to  be  enhanced  to 
meet  the  distributed  test  requirements. 

Requirements  need  to  be  allocated  to  the  participants  before  the  architecture  can  be  designed  and 
before  the  requirements  for  modifying  existing  player  representations  or  designing  new  ones  can 
be  determined.  These  allocated  requirements  include  the  following: 

•  Data  requirements. 

Determine  the  data  exchange  rates  among  the  players. 

Determine  the  time-space-position  information  (TSPI)  accuracy  and  smoothness 
requirements  for  live  players.  This  determination  depends  on  the  test  objectives  and  the 
data  input  requirements  of  the  player  receiving  the  TSPI  data. 

Determine  the  requirement  for  data  time  stamp  accuracy.  If  latency  is  to  be  measured,  the 
time  stamps  must  be  accurate  to  the  required  latency  determination  accuracy.  For  most 
applications,  an  accuracy  of  0.1  milliseconds  is  more  than  adequate.  Note  that  the,  time 
sources  which  determine  the  time  stamps  at  distributed  locations  must  be  synchronized  to 
within  the  required  time  stamp  accuracy. 

Determine  the  classification  of  the  data  and  any  security  handling  requirements.  This  is 
generally  driven  by  the  SUT  security  classification  guide. 

Document  all  data  exchanges  with  an  interface  control  document  (ICD). 

•  Data  synchronization  requirements.  Determine  the  requirement  for  synchronizing  multiple 
data  inputs  at  a  receiving  node. 

•  Real-time  data  processing  requirements.  These  might  include  the  processing  needed  to 
achieve  the  required  TSPI  accuracy  for  live  player,  telemetry  processing,  processing  needed 
for  synchronization  of  multiple  data  inputs  at  a  receiving  node,  and  processing  needed  to 
achieve  latency  compensation. 

•  Data  collection/instrumentation  requirements.  This  includes  data  which  must  be  recorded  for 
support  of  post-test  analysis. 


4 


These  allocated  requirements  are  used  to  assess  the  capabilities  of  the  player  representations 
selected  or  to  determine  design  requirements  for  any  missing  representations.  This  assessment 
may  reveal  the  need  to  modify  the  selected  representations.  Requirements  for  potential 
modifications  are  determined  as  follows: 

•  Determine  if  any  simulation  modifications  will  be  necessary  to  utilize  external  inputs. 

•  Determine  if  any  simulation  modifications  will  be  necessary  to  generate  required  outputs. 

•  Determine  if  any  range  data  processing  modifications  will  be  necessary  to  meet  TSPI 
accuracy,  smoothness,  and  latency  requirements. 

•  Determine  if  any  facility  modifications  will  be  required  for  a  replay  capability  which  can  be 
used  during  integration  testing. 

If  missing  player  representations  must  be  developed,  the  additional  requirements  are  used  for  the 
design  requirements.  The  decision  to  design  and  develop  a  new  representation  can  have  severe 
cost  and  schedule  impacts  on  the  distributed  test,  since  this  can  be  the  single  largest  cost  of  an 
ADS-based  test.  For  example,  the  JADS  End-to-End  (ETE)  Test  needed  a  ground  emulation  of 
the  Joint  Surveillance  Target  Attack  Radar  System  aircraft  radar  subsystem  and  operator  work 
stations,  and  no  adequate  simulation  existed.  Thus,  the  decision  was  made  to  develop  the 
required  simulation,  and  its  cost  was  about  40%  of  the  total  ETE  Test  budget. 

PREPARE  PLAN 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  a  coordinated  plan  to 
guide  the  development,  test,  and  execution  of  the  distributed  test.  The  following  documents  are 
needed: 

•  A  detailed  test  plan  which  includes  a  configuration  management  plan. 

•  A  detailed  DMAP  which  specifies  the  data  requirements,  data  sources,  analysis  procedures, 
and  the  analysis  products  required  to  accomplish  each  test  objective. 

•  A  detailed  VV&A  plan  which  includes  integration  testing  to  verify  architecture  performance. 

•  A  detailed  ICD  which  specifies  all  data  to  be  passed  between  simulations/range  facilities.  This 
data  specification  includes  data  content,  timing,  formats,  and  coordinate  systems  (including  all 
coordinate  transformation  equations).  The  ICD  is  essential  for  ensuring  successful  integration 
of  the  simulations/range  facilities  and  must  be  rigorously  enforced. 

ESTABLISH  PARTICIPANT  AGREEMENTS 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  establish  all  agreements  between 
participants  necessary  for  a  fully  consistent,  interoperable,  distributed  simulation  environment. 
During  this  activity,  the  participant  interaction  requirements  are  finalized,  including  the  following: 

•  Determine  the  data  protocols  to  be  used. 

Decide  if  standard  protocols  (e.g.,  DIS  PDUs)  are  to  be  used  or  if  it  would  be 
advantageous  to  keep  data  in  formats/coordinate  systems  generated  by  the  players. 

Identify  any  data  exchanged  between  players  which  is  best  kept  in  its  tactical  protocol. 

•  Interface  requirements.  Note  that  linking  of  facilities  using  ADS  can  require  significant 
interface  hardware  and  software  development.  ADS  implementation  is  not  “plug  and  play.” 
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-  Network  interface  units  (NIUs)  of  some  sort  are  necessary  if  two  simulations  cannot 
communicate  directly  in  a  common  language  and  on  a  common  timeline.  Determine 
coordinate  transformation  and  dead  reckoning  requirements. 

If  HLA  is  to  be  implemented,  interfaces  will  be  required  between  all  federates  (e.g.,  player 
representations,  range  facilities,  etc.)  and  the  runtime  infrastructure  (RTI). 

Operational  issues  and  policies  are  also  addressed  and  resolved,  including  the  following: 

•  Terrain  data  base  requirements.  Determine  requirements  for  a  common  terrain  data  base, 
including  resolution  and  level  of  detail. 

•  Post-test  data  management  requirements.  Determine  the  source  and  quantity  of  all  data  to  be 
recorded  at  each  recording  location.  A  distributed  test  with  numerous  trials  can  generate  a 
large  volume  of  data  at  distributed  locations.  Without  careful  planning,  key  data  may  not  be 
collected  and/or  transmitted  to  the  analysis  center,  and  data  collected  at  the  sites  may  not  be  in 
a  useful  form  for  centralized  analysis.  An  efficient  method  for  retrieving  the  data  to  a  central 
analysis  facility  is  by  use  of  the  network  links. 

•  Test  control  and  monitoring  requirements. 

-  Develop  the  test  control  concept,  including  a  determination  of  the  central  control  location 
and  the  test  coordination  location  at  each  distributed  node/facility.  Determine  the 
techniques  to  be  used  for  control  of  any  live  players. 

Determine  the  display  and  monitoring  requirements. 

Determine  the  voice  communications  requirements  for  effective  control  and 
monitoring  of  the  distributed  test. 

PLAN  EXECUTION 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  define  and  develop  the  full  set  of 
information  required  to  support  the  distributed  test  execution.  Detailed  network  design  should  be 
completed  during  this  activity,  although  design  work  should  begin  as  soon  as  the  participant 
allocated  requirements  have  been  defined.  Early  definition  of  network  requirements  allows  for 
timely  selection  of  hardware  and  software  alternatives  and  allows  enough  time  to  acquire  the  right 
components  through  government  channels  and  contracts.  Key  considerations  in  finalizing  the 
network  design  include  the  following: 

•  Determine  if  data  from  each  node  will  be  broadcast,  multicast,  or  unicast  (transmitted  point- 
to-point). 

•  Determine  if  data  are  to  be  transmitted  using  best  effort  or  reliable  procedures,  based  on  data 
transport  reliability  requirements. 

•  Determine  the  network  security  approach  to  be  implemented.  Detailed  procedures  for 
secure/encrypted  operations  must  be  developed,  and  approval  for  their  use  must  be  obtained 
from  the  designated  approval  authority. 

•  Determine  the  WAN  bandwidth  requirement. 

•  Conduct  surveys  of  each  site  to  be  linked  by  the  network. 

The  test  and  W&A  plans  should  be  refined,  especially  the  integration  test  plan  section.  In 
refining  the  integration  test  plan,  step-by-step  systematic  integration  testing  procedures  need  to  be 
developed  and  should  address  the  following: 
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•  Procedures  for  verifying  any  simulation/range  facility  modifications  or  any  new  player 
representations  in  a  systematic  stand-alone  fashion. 

•  Procedures  for  initially  testing  each  WAN  link  separately.  Testing  should  begin  at  the  channel 
service  unit  (CSU)/data  service  unit  (DSU)  level  to  make  sure  that  communications  work  at 
the  lowest  level. 

•  Procedures  for  testing  each  simulation-to-simulation  connection  with  all  network  nodes 
connected. 

•  Procedures  for  testing  the  voice  communications  with  all  equipment  and  personnel  as  in  the 
actual  test. 

Also,  detailed  test  control  procedures  and  a  security  test  and  evaluation  plan  should  be  developed. 
The  test  control  procedures  should  include  checklists  covering  activities  for  24  hours  prior,  4 
hours  prior,  1  hour  prior,  during  the  mission,  and  post-test.  Existing  stand-alone  facility/range 
procedures  serve  as  a  starting  point  for  developing  the  checklists.  The  system  integrator  uses 
these  to  develop  new  checklists  which  interleave  the  activities  at  each  facility  and  include  those 
unique  to  the  distributed  test  environment. 

INTEGRATE  AND  TEST  ADS  ARCHITECTURE 

This  activity  combines  the  separate  FEDEP  activities  of  “Integrate  Federation”  and  “Test 
Federation”  because  of  the  close  connection  between  the  two.  An  iterative  “test-fix-test” 
approach  is  recommended,  so  that  the  integration  and  test  activities  become  closely  interrelated. 
According  to  the  FEDEP  model,  the  purpose  of  these  activities  is  to  bring  all  of  the  distributed 
test  participants  into  a  unifying  operating  environment  and  to  test  that  they  can  all  interoperate  to 
the  degree  required  to  achieve  the  test  objectives. 

The  ADS  test  architecture  is  installed  during  this  activity  using  the  following  steps: 

•  The  WAN  to  be  used  to  link  the  facilities  is  selected  and  procured.  This  includes  determining 
whether  a  DoD-sponsored  network  can  support  the  test  requirements  or  if  commercial  leased 
lines  must  be  used. 

•  The  network  hardware  to  be  used  is  selected,  procured,  and  installed,  including  routers, 
CSUs/DSUs,  mulitplexers,  encryptors,  etc. 

•  The  interfaces  necessary  for  linking  are  built/procured  in  accordance  with  the  requirements 
developed  during  Activity  4.2. 

•  Test  control  hardware  and  software  are  selected,  procured,  and  installed  based  on  the 
requirements  developed  in  the  previous  activity. 

•  Network  analysis/monitoring  tools  are  selected  and  procured  and/or  developed. 

Key  testing  steps  during  this  activity  include  the  following: 

•  Perform  compliance  testing,  as  specified  in  the  W&A  plan.  Test  each  facility/node 
individually  to  ensure  that  ADS  capability  and  any  required  modifications  (including  software) 
have  been  correctly  implemented. 

•  Perform  integration  testing,  as  specified  in  the  integration  test  plan. 

Check  out  interfaces  and  facility  modifications  with  linking  between  pairs  of  nodes. 

Baseline  the  performance  of  the  network  with  no  loading  from  the  simulations/players. 
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-  Test  performance  of  critical  portions  of  the  network  under  loading  representative  of  test 
conditions  to  be  used. 

-  Use  an  iterative  “test-fix-test”  approach,  including  replay  of  trials  to  diagnose 
problems  and  verify  fixes. 

•  Perform  risk  reduction  missions.  The  purpose  of  these  fully  linked  missions  is  to  exercise  all 
parts  of  the  distributed  test  to  ensure  that  they  operate  as  intended.  The  early  risk  reduction 
missions  are  invaluable  for  identifying  problems  before  the  actual  test.  The  later  missions  are 
used  to  verify  fixes  and  serve  as  rehearsals  for  the  formal  test  execution. 

•  Perform  validation,  as  specified  in  the  W&A  plan. 

SUMMARY 

Experience  and  lessons  learned  from  JADS  testing  have  been  used  to  develop  a  methodology  for 
the  implementation  of  ADS-based  testing.  This  paper  only  briefly  overviews  selected 
methodology  activities;  the  complete  detailed  implementation  methodology  is  available  from  the 
JADS  web  page  at  http://www.jads.abq.com  until  1  March  2001.  After  that  date,  refer  requests 
to  HQ  AFOTEC/HO,  8500  Gibson  Blvd  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558 
(phone:  505-846-2579),  or  the  JT&E  Program  Office  Technical  Library,  2001  North  Beauregard 
St.  Suite  800,  Alexandria,  Virginia  22311  (phone:  703-578-8222).  The  methodology  has  been 
designed  as  a  practical  and  useful  tool  for  future  users  of  this  technology  and  constitutes  a  major 
part  of  the  legacy  that  JADS  will  transition  to  the  T&E  community. 
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